The present invention relates to an image interpolation method, image processing method, and image display method of interpolating and generating images at the viewpoint position of a user from existing images, apparatuses therefor, and computer program storage medium.
When a large-scale virtual environment is to be constructed, a technique based on real images is more advantageous in reality than geometric-model-based CG representation. Real images at a plurality of viewpoints are obtained by image-picking up an environment at a plurality of positions by using a camera having a position/posture sensor. Virtual images at arbitrary viewpoints are generated from these real images by using, for example, a morphing technique. In other words, the use of the morphing technique allows a user to experience a virtual space at arbitrary viewpoint positions beyond the viewpoint positions (assume that viewpoint positions include line of sight directions in the following description) at which image-picking up was performed.
In morphing, first of all, areas of two original images (e.g., images A and B) in which images of the same object are presented are made to correspond to each other in the form of, e.g., rectangles. That is, two partial image areas (e.g., areas Axe2x80x2 and Bxe2x80x2) recognized by the user as identical areas are set as corresponding areas. The shapes and colors of the insides of the two areas are linearly interpolated to construct a virtual image Q. As described above, linear interpolation is used for the generation of this virtual image Q. With regard to this linear interpolation, it is proposed that both the optical axes of the cameras at viewpoint positions PCA and PCB where the cameras sensed the two images A and B should be perpendicular to the base line (the straight line connecting the centers of the cameras) to effectively generate an interpolated image without any distortion.
S. M. Seitz and C. R. Dyer have proposed a technique of generating an interpolated image without any distortion even in a case wherein the optical axes at the two viewpoint positions PCA and PCB do not satisfy the above condition in xe2x80x9cView Morphingxe2x80x9d, Proc. SIGGRAPH ""96, pp. 21-30 (1996). The principle of this technique will be described below.
Assume that the images A and B are respectively obtained at the viewpoint positions PCA and PCB in FIG. 1. The optical axes of the cameras at the two viewpoint positions are neither parallel to each other nor perpendicular to the base line. Referring to FIG. 1, coordinates x1 and x2 indicate corresponding points with respect to one target point. When the virtual image Q at an arbitrary viewpoint position Pe is generated by linear interpolation, distortion occurs. According to S. M. Seitz and C. R. Dyer, therefore, as shown in FIG. 2, the two images are rotated as if images were obtained by performing image sensing operation in a state wherein both the optical axes are rotated to become perpendicular to the base line, and linear interpolation is performed for images Axe2x80x2 and Bxe2x80x2 after the rotations, thereby obtaining a virtual image Qxe2x80x2. This virtual image has no distortion.
S. M. Seitz and C. R. Dyer, however, point out that one significant constraint is set for this technique in the prior art, and interpolation itself cannot be executed if this constraint is not satisfied. More specifically, as shown in FIG. 3, in a case wherein the field of view of the camera at one viewpoint position includes the field of view of the camera at the other viewpoint position, when the images are rotated to make the optical axes become perpendicular to the base line and parallel to each other, no corresponding points exist on the two images after the rotations. Therefore, an image having no distortion cannot be generated by linear interpolation.
It is an object of the present invention to provide an image interpolation method which can eliminate the above constraint and generate a virtual image even at an arbitrary viewpoint position by interpolation.
In order to achieve the above object, according to one aspect of the present invention, there is provided a method of generating an interpolated image at a user viewpoint position different from first and second viewpoint positions on the basis of first and second images obtained at the first and second viewpoint positions, is characterized by comprising:
the first rotation step of rotating the first and second images about the first and second viewpoint positions, respectively, such that optical axes of cameras which respectively took the first and second images coincide with a first direction substantially coinciding with a direction of a base line connecting the first and second viewpoint positions;
the first coordinate interpolation step of calculating coordinates of a corresponding point of an interpolated image having the first direction as a line of sight direction at the user viewpoint position by performing nonlinear proportional distribution for coordinates of a first corresponding point in the first image with respect to a predetermined feature point and coordinates of a second corresponding point in the second image with respect to the feature point; and
the second rotation step of rotating the interpolated image from the first direction to the line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.
According to this interpolation method, by making the optical axes of two images coincide with each other, nonlinear proportional distribution can be performed, and an interpolated image at the user viewpoint position can be generated. If this interpolated image is rotated in accordance with the line of sight direction of the user, the rotated image becomes an image interpolated in the line of sight direction at the user viewpoint position.
The above nonlinear proportional distribution processing should be performed for only images satisfying a predetermined relationship. According to another aspect of the present invention, therefore, the method further comprises the determination step of determining whether a field of view of the first image includes a lens center of the second image or a field of view of the second image includes a lens center of the first image, and the first rotation step is executed when YES is obtained in the determination step.
According to yet another aspect of the present invention, the method is characterized by further comprising: the determination step of determining whether a field of view of the first image includes a lens center of the second image or a field of view of the second image includes a lens center of the first image;
the third rotation step of rotating the first and second images about the first and second viewpoint positions, respectively, such that optical axes of the images become substantially perpendicular to the direction of the base line, when NO is obtained in the determination step;
the second coordinate interpolation step of calculating coordinates of a corresponding point of an interpolated image having the second direction perpendicular to the direction of the base line at the user viewpoint position as a line of sight direction by performing linear proportional distribution for coordinates of a first corresponding point of the first image and coordinates of a second corresponding point of the second image; and
the fourth rotation step of rotating the interpolated image from the second direction to the line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.
According to still another aspect of the present invention, the method is characterized in that when the user viewpoint position internally divides an interval between the first and second viewpoint positions at t:1xe2x88x92t, the nonlinear proportional distribution processing in the first coordinate interpolation step is executed by making a reciprocal value 1/xe of a coordinate of the interpolated image divide an interval between a reciprocal value 1/xA of the coordinate of the first corresponding point and a reciprocal value 1/xB of the coordinate of the second corresponding point at t:1xe2x88x92t.
According to still another aspect of the present invention, the method is characterized in that when the user viewpoint position internally divides an interval between the first and second viewpoint positions at t:1xe2x88x92t, the linear proportional distribution processing in the second coordinate interpolation processing is executed by making a coordinate value xe of the interpolated image divide an interval between a coordinate value xA of the first corresponding point and a coordinate value xB of the second corresponding point at t:1xe2x88x92t.
The above nonlinear interpolation processing should be performed for images satisfying a predetermined relationship. According to still another aspect of the present invention, therefore, the method further comprises the step of determining whether the first and second images have at least image data of corresponding points derived from the same object.
According to still another aspect of the present invention, the method further comprises the step of searching an image database for another image having at least a corresponding point when the first and second images do not have at least image data of corresponding points derived from the same object.
According to still another aspect of the present invention, the first and second viewpoint positions are defined by viewpoint positions and optical axis directions of the cameras in image sensing operation, and the user viewpoint position is defined by a viewpoint position and line of sight direction of the user.
According to still another aspect of the present invention, the interpolated image output in the first or fourth rotation step is presented as a panoramic image to the user.
It is another object of the present invention to provide an image interpolation method which has no limitation to allow a user to move in a wide range.
In order to achieve this object, according to another aspect of the present invention, there is provided a method of generating an interpolated image at a user viewpoint position different from first, second, and third viewpoint positions on the basis of first, second, and third images respectively obtained at the first, second, and third viewpoint positions, characterized by comprising:
the determination step of determining whether any two of the first to third images have an inclusive relationship;
internal ratio calculation step of obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;
the first coordinate interpolation step of performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;
the second coordinate interpolation step of performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;
the first image interpolation step of generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;
the second image interpolation step of generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship; and
the rotation step of rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.
According to this method, images can be interpolated regardless of whether the images have an inclusive relationship or not.
According to still another aspect of the present invention, the first coordinate interpolation step comprises the step of rotating the two arbitrary images at viewpoint positions in the base line direction to make optical axes of the images coincide with each other, and the interpolation processing is performed for the two rotated images.
According to still another aspect of the present invention, the method is characterized in that the second coordinate interpolation step comprises the step of rotating the two arbitrary images at viewpoint positions in a direction perpendicular to the base line direction to make the two arbitrary images become parallel to each other, and the interpolation processing is performed for the two rotated images.
According to still another aspect of the present invention, the method is characterized in that the determination step comprises determining whether a field of view of one image includes a lens center of the other image.
According to still another aspect of the present invention, the method is characterized in that a viewpoint position which divides, at a second internal ratio, an interval between the third viewpoint position and a temporary virtual position PT which divides an interval between the first and second viewpoint positions at a first internal ratio coincides with the user viewpoint position.
According to still another aspect of the present invention, the method is characterized in that the first coordinate interpolation step comprises obtaining a reciprocal value 1/xe of a coordinate of the temporary virtual image by internally dividing an interval between a reciprocal value 1/xA of a coordinate of a first corresponding point of one of the two images and a reciprocal value 1/xB of a coordinate of a second corresponding point of the other image at the first internal ratio.
According to still another aspect of the present invention, the method is characterized in that the second coordinate interpolation step comprises obtaining a coordinate value xe of the temporary virtual image by internally dividing an interval between a coordinate value xA of a first corresponding point of one of the two images and a coordinate value xB of a second corresponding point of the other image at the second internal ratio.
According to still another aspect of the present invention, the method further comprises the step of searching an image database for another image having at least a corresponding point when the first to third images do not have at least image data of corresponding points derived from the same object.
It is still another object of the present invention to provide an image processing apparatus which has no limitation on image interpolation and allows a user to move an image in a two-dimensionally wide space.
In order to achieve this problem, according to another aspect hereinafter aspect A of the present invention, there is provided an image processing apparatus characterized by comprising: an image database storing a plurality of images together with pieces of viewpoint position information corresponding thereto in advance;
means for detecting both a line of sight direction and viewpoint position of a user;
means for searching the image database for the first to third images including corresponding points in the line of sight direction of the user;
determination means for determining whether any two of the first to third images have an inclusive relationship;
internal ratio calculation means for obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;
first coordinate interpolation means for performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;
second coordinate interpolation means for performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;
first image interpolation means for generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;
second image interpolation means for generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship; and
rotation means for rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.
According to still another aspect of the present invention, the apparatus is characterized in that the first coordinate interpolation means comprises means for rotating the two arbitrary images at viewpoint positions in the base line direction to make optical axes of the images coincide with each other, and the interpolation processing is performed for the two rotated images.
According to still another aspect of the present invention, the apparatus is characterized in that the second coordinate interpolation means comprises means for rotating the two arbitrary images at viewpoint positions in a direction perpendicular to the base line direction to make the two arbitrary images become parallel to each other, and the interpolation processing is performed for the two rotated images.
According to still another aspect of the present invention, the apparatus is characterized by further comprising means for searching an image database for another image having at least a corresponding point when the first to third images do not have at least image data of corresponding points derived from the same object.
In order to expand the range in which the user can move an image, according to another aspect of the present invention, the apparatus is characterized by further comprising means for storing a temporary virtual image generated by the first image interpolation means in the image database, together with corresponding viewpoint position information.
In order to expand the range in which the user can move, according to still another aspect of the present invention, the apparatus is characterized by further comprising means for storing a temporary virtual image generated by the second image interpolation means in the image database, together with corresponding viewpoint position information.
It is still another object of the present invention to provide an image processing apparatus for processing an image for morphing processing, and more specifically, an image obtained by sensing an object having uneven portions into an image suited to morphing.
In the interpolation processing according to the preceding aspects, a user viewpoint position for interpolation is set between the first and second viewpoint positions (and the third viewpoint position) or an arbitrary position inside a given range. It is therefore expected that the relative front/rear relationship between sensed objects remains unchanged in original images suited to the interpolation processing in the preceding aspects even if the user viewpoint position is set at an arbitrary intermediate position between the first and second viewpoint positions (and the third viewpoint position) or an arbitrary position inside a given range. This means that original images suited to the interpolation processing in the preceding aspects must be generated in advance.
In order to achieve the above object, according to another aspect of the present invention, there is provided an image processing apparatus for generating a morphed image by processing first and second two-dimensional images respectively obtained by sensing an object at first and second viewpoint positions, characterized by comprising:
segmentation means for segmenting the first two-dimensional image into a plurality of first two-dimensional partial images in accordance with a plurality of depths from the first viewpoint position, and also segmenting the second two-dimensional image into a plurality of second two-dimensional partial images in accordance with a plurality of depths from the second viewpoint position.
The front/rear relationship between objects reflected in a plurality of two-dimensional partial images segmented and generated in accordance with depths is held at an arbitrary intermediate viewpoint position between the first and second viewpoint positions. When, therefore, such two-dimensional partial images are subjected to morphing processing afterward, morphing in which the front/rear relationship is correctly reflected is guaranteed. When the user sees two objects having a front/rear relationship while moving, part of one object must be occluded by part of the other object from time to time. If morphing is performed on the basis of the two images without giving any consideration to the front/rear relationship, an interpolated image is generated such that a portion which should not be seen can be seen or a portion which should be seen cannot be seen. Partial images generated by segmentation in consideration of the front/rear relationship between objects can be presented to the user such that portions which should be seen can be seen, and portions which should be occluded can be occluded. By visualizing an occluded portion which should be seen to the user, he/she can add image data (retouch operation) to the portion.
Two-dimensional images essentially have no depth information. Segmentation/generation of two-dimensional partial images from such two-dimensional images in accordance with depths must be easy and user-friendly. According to another aspect of the present invention, therefore, the segmentation means comprises:
display means for displaying the first and second two-dimensional images; and
first designation means for designating partial areas in the displayed first and second two-dimensional images as a pair of two-dimensional partial images by adding hierarchical information corresponding to a depth to each of the partial images.
The user can determine the front/rear relationship between objects in a displayed picture by seeing displayed images, and hence can correctly designate a partial area.
When morphing processing is performed from an arbitrary viewpoint position, an occluded portion needs to be set in the field of view in some case. For such a case, in the image processing apparatus according to another aspect of the present invention, the first designation means comprises:
second designation means for designating, on the display means, a partial area in which a two-dimensional partial image belonging to a layer farther from the arbitrary viewpoint position is occluded by a two-dimensional partial image belonging to a layer nearer to the arbitrary viewpoint position; and
retouch means for allowing a user to retouch a pixel value of the occluded partial area.
Even if an occluded partial area is set in the field of view as a result of morphing processing, since this area is retouched, no problem is posed.
According to another aspect of the present invention, the first designation means is characterized by comprising rewrite means for rewriting pixel values of one or a plurality of pairs of two-dimensional partial images having a second hierarchical information value other than a first hierarchical information value into a predetermined fixed value with respect to one pair of two-dimensional partial images having the first hierarchical information value. According to another aspect of the present invention, the predetermined fixed value is a pixel value representing black.
A partial image may include images of a plurality of objects located close to each other. According to still another aspect of the present invention, therefore, the segmentation means comprises third designation means for designating a plurality of partial areas (objects) in one pair of first and second two-dimensional partial images corresponding to a given layer.
Note that image data can be stored in various forms. According to still another aspect of the present invention, the segmentation means stores first and second original two-dimensional images and a plurality of pairs of two-dimensional partial images, together with hierarchical information.
The image processing apparatus according to the preceding seven aspects of the present invention is used to generate and store images suited to interpolation processing (morphing processing). Preferably, therefore, there is provided an image display apparatus for reading out a plurality of pairs of two-dimensional partial images generated and stored by the image processing apparatus defined in any of the preceding seven aspects, and performing morphing processing to generate an image from an arbitrary viewpoint position. The image display apparatus according to another aspect of the present invention (aspect B) is characterized by comprising:
first morphing processing means for performing, for pairs of two-dimensional partial images corresponding to all depths, processing of generating a partial morphed image from an arbitrary viewpoint position on the basis of one pair of first and second two-dimensional partial images corresponding to each depth; and
display means for synthesizing partial morphed images corresponding to the respective depths and generated by the first morphing processing means, and displaying a synthesized image on a predetermined display unit.
According to still another aspect of the present invention, the apparatus is characterized by being applied to a walkthrough application.
According to still another aspect of the present invention, the apparatus further comprises input means for inputting information about the arbitrary viewpoint position.
According to still another aspect of the present invention, the input means is characterized by virtually changing a viewpoint position value.
All images are not necessarily suited to morphing to be performed in consideration of depths. This is because morphing to be performed in consideration of depths takes much time. The image display apparatus according to another aspect is therefore characterized by further comprising: second morphing processing means for generating a morphed image directly based on the first and second two-dimensional images; and
switching means for selectively switching the first morphing processing means and the second morphing processing means.
The image display apparatus according to still another aspect of the present invention provides practical switching means and is characterized in that the apparatus further comprises:
input means for inputting information about the arbitrary viewpoint position; and
a memory for determining that one of the first morphing processing and the second morphing processing is to be applied to an arbitrary walkthrough route in a walkthrough virtual space when a viewpoint position of the user exists in the walkthrough route, and storing information indicating a determination result together with a position of the walkthrough route, and
the switching means extracts determination result information corresponding to viewpoint position information input by the input means from the memory, and selects and operates one of the first morphing processing means and the second morphing processing means on the basis of the information.
The image display apparatus according to aspect B of the present invention can be effectively combined with the image processing apparatus according to aspect A of the present invention. The image display apparatus according to this aspect to the present invention is characterized in that the first morphing means comprises:
an image database storing a plurality of images together with pieces of viewpoint position information corresponding thereto in advance;
means for detecting both a line of sight direction and viewpoint position of a user;
means for searching the image database for the first to third images including corresponding points in the line of sight direction of the user;
determination means for determining whether any two of the first to third images have an inclusive relationship;
internal ratio calculation means for obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;
first coordinate interpolation means for performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;
second coordinate interpolation means for performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;
first image interpolation means for generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;
second image interpolation means for generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship; and
rotation means for rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.